IMAGE PROCESSOR, 
METHOD OF PROVIDING IMAGE PROCESSING SERVICES AND 
ORDER PROCESSING METHOD 

« 

5 BACKGROUND OF THE INVENTION 

The present invention generally relates to image proc- 
essing technology, and more particularly relates to a tech- 
nique of processing an image by utilizing information about 
O the three-dimensional position of an object. 

Ni 10 FIG. 8 illustrates a basic arrangement for a rangefinder 

J; that can capture a range image (or depth image). The range- 

^ finder shown in FIG. 8 pertains to the art closely related to 

the present invention and is disclosed by us in Japanese Pat- 
ent Application No. 11-144097, which is hereby incorporated by 
p 15 reference. As shown in FIG. 8, the rangefinder includes cam- 
era 51, light sources 52a and 52b, light source controller 55 
and distance calculator 56. In response to a vertical sync 
signal supplied from the camera 51, the light source control- 
ler 55 gets each of the light sources 52a and 52b alternately 
20 flashed. The distance calculator 56 generates a range image 
from an image captured by the camera 51. 

FIG. 9A is a perspective view illustrating exemplary con- 
figurations for the light sources 52a and 52b. As shown in 
FIG. 9A, flash lamps 57 and 58 such as xenon lamps are verti- 
25 cally stacked in the light sources 52a and 52b, respectively. 
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Reflectors 59 and 60 are placed behind these lamps 57 and 58, 
respectively, so that the range of the light reflected off 
from one of these reflectors 59 and 60 horizontally shifts 
from that of the light reflected off from the other. FIG. 9B 
5 is a plan view of the light sources 52a and 52b shown in FIG. 
9A. As shown in FIG. 9B, the light sources 52a and 52b radi- 
ate, or project, light beams within the ranges A and B, re- 
spectively. In the illustrated example, the xenon lamps have 
so small emissive portions that these lamps can be virtually 

10 regarded as point light sources in the plan view. Also, the 
light sources 52a and 52b are vertically spaced apart from 
each other by about 1 cm. Accordingly, the light may be re- 
garded as being emitted from almost a point. 

Hereinafter, the operating principle of the rangefinder 

15 shown in FIG. 8 will be described with reference to FIGS. 10 
through 13. 

FIG. 10 schematically illustrates exemplary light pat- 
terns that have been radiated from the light sources 52a and 
52b shown in FIG. 9. In FIG. 10, the solid lines La and Lb 

20 represent the brightness on a virtual screen Y, on which the 
light beams have been projected from the light sources 52a and 
52b. The higher the solid lines La and Lb in the direction 
indicated by the arrow in FIG. 10, the brighter the light pro- 
jected. As can be seen from FIG. 10, the light projected from 

25 each of these light sources 52a and 52b is intensest, or 
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brightest/ on the center axis of the projection range and gets 
gradually weaker, or darker, toward the edges of the range. A 
distribution like this results from the disposition of the 
semi-cylindrical reflectors 59 and 60 behind the flash lamps 
5 57 and 58, respectively. And depending on which directions 
these reflectors 59 and 60 face, the light beams projected 
from the light sources 52a and 52b may or may not partially 
overlap with each other. 

FIG. 11 is a graph illustrating a relationship between 

10 the angle (t> of the projected light as measured in the direc- 
tion H shown in FIG. 10 and the light intensity. In this case, 
the direction H is defined as a direction in which an arbi- 
trary plane S, including the respective center of the light 
source and lens, and the virtual screen Y intersect with each 

15 other. The angle (J> is an angle formed by the light, which 
has been projected onto the XZ plane, with the X-axis. As 
shown in FIG. 11, a range a of the light pattern projected 
from one of the light sources 52a and 52b through the object's 
space has relatively bright and relatively dark parts on 

20 right- and left-hand sides of the light source 52a or 52b, re- 
spectively. Conversely, the range a of the light pattern 
projected from the other light source 52b or 52a through the 
object's space has relatively dark and relatively bright parts 
on right- and left-hand sides of the light source 52b or 52a, 

25 respectively. It should be noted that the light patterns 
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shown in FIG, 11 change in the height direction (i.e., the Y 
direction). In other words, the patterns are changeable de- 
pending on the level at which the plane including the centers 
of the light sources and lens is located. 
5 FIG. 12 is a graph illustrating a relationship between 

the angle 4> of the projected light and the light intensity 
ratio in the range a shown in FIG. 11. As shown in FIG. 12, 
there is a substantially linear relationship between the light 
intensity ratio and the angle 0 in the range a. 

10 To measure the distance of an object, two types of light 

patterns should be alternately projected in advance onto a 
plane standing vertically at a predetermined distance from the 
light source and light beams; reflected from the plane should 
be received and imaged at the camera 51. A relationship be- 

15 tween the light intensity ratio and the angle of the projected 
light such as that shown in FIG. 12 should be obtained in ad- 
vance for each Y coordinate (corresponding to a Y coordinate 
on the CCD). And the light sources 52a and 52b should be dis- 
posed so that a line connecting the center of the camera lens 

20 to the light sources 52a and 52b is parallel to the X-axis of 
the CCD imaging plane. In that case, the distance of the ob- 
ject can be estimated accurately based on the data represent- 
ing the relationships between the light intensity ratios and 
angles of the projected light associated with the respective Y 

25 coordinates obtained beforehand. 
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Now, take a look at the point P shown in FIG. 8. First, 
the intensity ratio of the light beams projected from the 
light sources 52a and 52b onto the point P is obtained from 
the image captured by the camera 51, And the angle 0 corre- 
5 spending to the point P as viewed from the light sources 52a 
and 52b can be derived from the resultant intensity ratio by 
reference to the relationship shown in FIG. 12 associated with 
the Y coordinate of the point P. Also, the angle 9 formed by 
a visual axis, which extends from the center of the camera 51 

10 to the point P, with the X-axis can be obtained based on the 
coordinates of a pixel associated with the point P and various 
camera parameters including coordinates of the optical center 
of the lens system. Then, the distance of the point P is es- 
timated by the triangulation technique using the two angles 

15 obtained 0 and (t> and a baseline length D, i.e., the distance 
between the position of the light sources 52a and 52b and the 
optical center of the camera 51. 

Suppose the optical center of the camera 51 is defined 
as the origin O of the coordinate system; the optical axis of 

20 the camera 51 as the Z-axis thereof; horizontal and vertical 
directions as the X- and Y-axes thereof; an angle formed by a 
visual axis extending from the light sources 52a and 52b to 
the point P with the X-axis as 0 ; an angle formed by a visu- 
al axis extending from the camera 51 to the point P with the 

25 X-axis as 0 ; and the coordinates of the light sources 52a 
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and 52b are (0, -D) (i.e., the baseline length is D). Then, 
the depth Z of the point P is given by 

Z =Dtan0 tan / (tan^ — tan<^ ) 
Alternatively, all of the three-dimensional coordinates (X, Y, 
5 Z) of the point P may be calculated by the following equa- 
tions using the angle a; shown in FIG* 13: 

X = Z/tan0 

Y = Z/tana) 

Furthermore, an ordinary color image can also be obtained 
10 by adding and averaging the images formed by flashing the 
light sources 52a and 52b, Accordingly, an image containing 
three-dimensional (3D) positional information can be captured 
by using the arrangement shown in FIG. 8. 

However, even though it is now technically possible to 
15 capture an image with 3D positional information in this manner, 
the technique per se does not automatically allow each and 
every maker to market attractive, potentially big-hit products. 
Accordingly, in developing consumer electronic products, it is 
very important for the makers to add highly convenient and en- 
20 tertaining functions to the products by taking advantage of 
the technique. 



SUMMARY OF THE INVENTION 

It is therefore an object of the present invention to 
25 provide highly user-friendly and interesting functions by 
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utilizing 3D positional infoirmation of an object in image 
processing. 

Specifically, an inventive image processor includes: a 
display for presenting an image of an object thereon; and an 
5 image synthesizer for generating a scale image, representing 
a substantially real size, at a position specified on the im- 
age presented on the display in accordance with 3D positional 
information of the object and for combining the scale image 
with the image of the object. A synthesized image, obtained 

10 by combining the scale image with the object image, is pre- 
sented on the display. 

According to the present invention, a scale image, rep- 
resenting a substantially real size, is combined at a speci- 
fied position with an image originally presented on the 

15 display so that a synthesized image is presented on the dis- 
play. Thus, the user can know a real size of the object im- 
aged quickly on looking at the synthesized image presented. 

In one embodiment of the present invention, the proces- 
sor may further include: an imaging section for capturing the 

20 object image containing the 3D positional information; and a 
range image generator for drawing the 3D positional informa- 
tion from the image captured by the imaging section. The im- 
age synthesizer preferably generates the scale image in 
accordance with the 3D positional information obtained by the 

25 range image generator. More specifically, the imaging sec- 
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tion preferably includes light-emitting means for projecting 
light with a predetermined radiation pattern onto the object 
and captures the object image containing the 3D positional in- 
formation by receiving part of the light that has been pro- 
5 jected onto, and then reflected from, the object. 

In an alternative embodiment, the processor may further 
include an imaging section having an automatic or manual fo- 
cusing controller. The image synthesizer preferably generates 
the scale image by using data, which represents a distance of 

10 the object and is obtained by the automatic or manual focus- 
ing controller, as the 3D positional information. 

In still another embodiment, the processor may further 
include input means that is so constructed as to allow a user 
to externally input the specified position. Specifically, the 

15 input means may be a touch panel formed on the surface of the 
display, a penlike pointing device so constructed as to allow 
the user to specify arbitrary coordinates on the surface of 
the display or a cursor key, mouse or press button that allows 
the user to move a cursor presented on the display and to 

20 specify coordinates of the cursor. 

Another inventive jLmage processor includes: a display 
for presenting an image of an object thereon; and an image 
synthesizer for combining respective images of multiple ob- 
jects together in accordance with 3D positional information 

25 of the objects so that at least one of the object images is 
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scaled up or down according to a desired size relationship. 
A synthesized image, obtained by combining the multiple im- 
ages together, is presented on the display. 

According to the present invention, multiple images are 
5 combined together after at least one of the images has been 
appropriately scaled up or down according to a user defined 
size relationship (e.g., to meet an actual size relationship) 
and then a resultant synthesized image is presented on the 
display. Thus, the user can combine an image of the object 

10 with another background image with their scales matched. In 
this manner, the user can quickly watch a virtual image of 
the object on his or her favorite non-real background. 

In one embodiment of the present invention, the image 
synthesizer may combine the image of one of the objects, which 

15 has been separated from a background image, with another back- 
ground image. In this particular embodiment, the image syn- 
thesizer preferably cuts out an image portion, which is made 
up of pixels at respective locations associated with distances 
falling within a predetermined range, as the separated object 

20 image from the linage. 

In an alternative embodiment, the processor may further 
include: an imaging section for capturing the object images 
containing the 3D positional information; and ^a range image 
generator for drawing the 3D positional information from the 

25 images captured by the imaging section. The image synthesizer 
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may combine the images together in accordance with the 3D 
positional information obtained by the range image generator. 
More specifically, the imaging section preferably includes 
light-emitting means for projecting light with a predetermined 
5 radiation pattern onto the at least one object and captures 
the object image containing the 3D positional information by 
receiving part of the light that has been projected onto, and 
then reflected from, the object. 

In still another embodiment, the processor may further 

10 include an imaging section having an automatic or manual fo- 
cusing controller. The image synthesizer may combine the im- 
ages together by using data, which represents distances of 
the objects and is obtained by the automatic or manual focus- 
ing controller, as the 3D positional information. 

15 Still another inventive image processor includes a dis- 

play for presenting an image of an object thereon and an im- 
age synthesizer for generating an image, representing the 
object substantially in its real size when presented on the 
display, by scaling the image up or down in accordance with 

20 3D positional information of the object. The image, repre- 
senting the object substantially in its real size, is pre- 
sented on the display. 

An inventive method of providing image processing servi- 
ces for a user includes the steps of: receiving an image, 

25 containing 3D positional information of an object, and his or 
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her specified image processing mode from the user; drawing 
the 3D positional information from the object image received; 
performing image processing on the received image in the 
specified image processing mode in accordance with the 3D 
5 positional information drawn; and sending out image data ob- 
tained by the image processing to the user. 

An inventive order processing method includes the steps 
of: receiving an instruction, which specifies a type of a 
product, from a user; sending out an image, containing 3D 
10 positional information of the product of the specified type, 
to the user; and receiving an order for the product, along 
with information specifying his or her desired size of the 
product, from the user. 



15 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG, 1 is a block diagram illustrating an arrangement 
for a geometric measuring camera as an exemplary image proc- 
essor according to first and second embodiments of the pre- 
sent invention, 

20 FIG. 2 is a perspective view illustrating the back ap- 

pearance of the camera shown in FIG. 1. 

FIGS, 3A through 3F illustrate how the camera of the 
first embodiment operates - 

FIG, 4 is a plan view illustrating the back of a camera 
25 equipped with cursor keys . 



FIG. 5 is a block diagram illustrating an arrangement 
for a camera including an image memory. 

FIGS. 6A through 6E illustrate how a camera according to 
the second embodiment operates. 
5 FIG. 7 illustrates perspective transformation. 

FIG. 8 illustrates a basic arrangement for a rangefinder 
that can capture a range image. 

FIGS. 9A and 9B are respectively perspective view and 
plan view illustrating the light sources shown in FIG. 8. 
10 FIG. 10 illustrates light patterns formed by the light 

sources shown in FIGS. 9A and 9B. 

FIG. 11 is a graph illustrating a relationship between 
the light intensity and the angle of projected light for the 
light patterns shown in FIG. 10. 
15 FIG. 12 is a graph illustrating a relationship between 

the intensity ratio and the angle of projected light for the 
range a shown in FIG. 11. 

FIG. 13 schematically illustrates the concept of an an- 
gle CO used for obtaining 3D coordinates (X^ Y, Z). 
20 FIG. 14 is a block diagram schematically illustrating 

image processing services according to the present invention. 

FIG. 15 is a block diagram schematically illustrating a 
system for implementing an order processing method according 
to the present invention. 

25 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, preferred embodiments of the present inven- 
tion will be described with reference to the accompanying 
drawings . 

5 As used herein, a "range image" is an image containing 

data about either distances associated with respective pixels 
as measured from, a camera or depth values in the three- 
dimensional coordinate system. Each of these distances cor- 
responds to a spherical coordinate r of the spherical coordi- 

10 nate system (r, 0 , <t> ) , while each of the depth values 
corresponds to an orthogonal coordinate z of the orthogonal 
coordinate system (x, y/.z)- In a first embodiment of the 
present invention, the spherical coordinate r will be used. 
On the other hand, the orthogonal coordinate z will be used 

15 in a second embodiment of the present invention. It should 
be noted, however, that the spherical coordinate system (r, 
9 , 0) and orthogonal coordinate system (x, y, z) are mutual- 
ly convertible bidirectionally - Also, by using a pixel loca- 
tion with two-dimensional coordinates (x, y) on a CCD, 

20 spherical coordinates (r, 9 , (/> ) and orthogonal coordinates 
(x, y, z) can be easily obtained from r and z, respectively, 
through three-dimensional geometric computations. 

EMBODIMENT 1 

25 FIG. 1 illustrates an arrangement for a geometric meas- 
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uring camera as an exemplary image processor according to the 
first embodiment. As shown in FIG. 1, the camera includes 
casing 1, first and second flash lamps 11a and lib, light 
source controller 12, imager 13, range image generator 14 and 
5 color image generator 15. The first and second flash lamps 
11a and lib are used as respective light sources. The light 
source controller 12 controls flashing of the first and sec- 
ond flash lamps 11a and lib. The range image generator 14 
generates a range image from the image captured by the imager 

10 13. The range image contains data about the distances associ- 
ated with respective pixel locations on the image as measured 
from the camera. And the color image generator 15 generates a 
normal color image from the image captured by the imager 13, 
The first and second flash lamps 11a and lib and light source 

15 controller 12 together constitutes light-emitting means 16. 
And the imager 13 and light-emitting means 16 together makes 
up an imaging section 10. The light-emitting means 16 is re- 
movable from the body, i.e., the imaging section 10. 

The camera further includes display panel 21, touch panel 

20 22, shutter release 23 and recording/reproducing section 24. 
The display panel 21 is provided to present an image thereon. 
The touch panel 22 is formed on the surface of the panel 21. 
And the recording/reproducing section 24 writes or reads out 
the range image or color image onto/from a storage medium 25. 

25 The camera further includes a control unit 30 with a 
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built-in image synthesizer 31. Responsive to an output signal 
of the shutter release 23, the control unit 30 not only con- 
trols the operations of the light source controller 12 and 
imager 13 but also provides the input color image to the dis- 
5 play panel 21 so that the image will be presented thereon. 
The image synthesizer 31 combines a scale image, representing 
a real size at a specified position on the image, with the im- 
age itself. 

In the camera shown in FIG. 1, the imager 13 can capture 
10 an image, including 3D positional information, as in the exam- 
ple described in the background section. In addition, accord- 
ing to this embodiment, by using the range image, generated by 
the range image generator 14, as the 3D positional informa- 
tion, a scale image, representing a substantially real size, 
15 can be presented on the panel 21 along with a normal color im- 
age. 

FIG. 2 is a perspective view schematically illustrating 
the back appearance of the camera shown in FIG. 1. As shown 
in FIG. 2, the display and touch panels 21 and 22 are stacked 

20 in this order on the backside of the camera 1. By manipulat- 
ing the touch panel 22, the user can specify his or her de- 
sired position on the image, where he or she wants the scale 
image to be presented on the display panel 21, so that he or 
she can know a real size of the object. 

25 Hereinafter, it will be described with reference to 



15 




FIGS. 3A through 3F how the geometric measuring camera of the 
first embodiment operates in such a manner as to present a 
scale image in accordance with the user's instructions. 

First, as shown in FIG. 3A, the user takes a picture of 
5 an object (e.g., a desk placed in a room in the illustrated 
example) with the camera of this embodiment. In this case, 
the image generated includes 3D positional information of re- 
spective parts of the desk. Then, the user has a color image 
of the desk presented on the display panel 21. 

10 Next, as shown in FIG. 3B, the user specifies his or her 

desired position on the image, at which the real size should 
be represented by the scale image. The user may select a 
scale display mode for the display panel 21 by turning ON a 
mode switch provided for the body of the camera, for instance. 

15 Then, the user may specify a position C (i.e., a point on the 
upper surface of the desk in the illustrated example) using 
the touch panel 22 while watching the screen. 

In response, the image synthesizer 31 calculates a real 
size for the specified position C based on the range image 

20 thereof, generates a scale image representing the real size 
obtained and then combines the scale image with the color im- 
age. As a result, the scale image S, representing the real 
size of the upper surface of the desk, is presented at the 
specified position C as shown in FIG. 3C. In the illustrated 

25 example, the scale image S is an image representing a ruler 
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with a scale. 

The real size may be calculated in the following manner. 
Specifically, based on a Euclidean distance between the 
specified position C with a set of three-dimensional coordi- 
5 nates and the center of the camera lens with a different set 
of three-dimensional coordinates, the distance L between the 
camera and the object is derived. The real length repre- 
sented by each pixel on the screen for the specified position 
C is given by 

10 P^ = L/f • S^/N^ 

where f is the focal length of the camera, is a CCD size 
of the camera and is the number of horizontal active pix- 
els in the CCD. Accordingly, the number Np of on-screen pix- 
els equivalent to a real length X is given by 

15 Np = X/P^ 

Thus, the scale image S with a length represented by the pix- 
el number is combined with letters L representing the real 
length X (e.g., 2 0 cm in the illustrated example) at the 
specified position C. 

20 Also, as shown in FIGS. 3D and 3E, the user may change 

the direction and display position of the presented scale im- 
age S by manipulating the touch panel 22. It should be noted 
that when the far end of the scale image S presented is di- 
rected toward the depth of the screen as shown in FIG. 3D, 

25 the di3t:ance L between the camera and the object changes 
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gradually. Accordingly, the scale image S will have a non- 
even (i.e., diminishing) scale in that case. Optionally, the 
processor may also be constructed so that the user can select 
an arbitrary one of multiple scale images S with mutually 
5 different scales. 

If the user wants to remove the scale image S from the 
screen, then he or she cancels the scale display mode by 
turning the mode switch OFF, for example. As a result, the 
scale image S disappears from the screen and only the origi- 

10 nal color image is left as shown in FIG. 3F. 

In this manner, according to this embodiment, the user 
can easily know a real size of an imaged object by reference 
to the scale image, representing the real size on the screen, 
while watching the image. Thus, on looking at an image of an 

15 object, e.g., a fish on a fishhook or a finding that has been 
dug up in some remains, the user can know its real size 
quickly and easily without placing any size reference object 
beside it. Also, if the camera of this embodiment is used as 
a security camera, then the user can easily identify a body 

20 size of a criminal imaged. 

In the first embodiment, the specified position C is 
supposed to be selected using the touch panel 22. Alterna- 
tively, a penlike pointing device, allowing the user to 
specify arbitrary coordinates on the display panel 21, may be 

25 used instead of the touch panel 22. Cursor keys, press but- 
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tonS/ dial or mouse/ which allows the user to move a cursor on 
the display panel 21, may also be used. FIG. 4 is a plan 
view illustrating the back of the camera 1 equipped with cur- 
sor keys 28. In the example illustrated in FIG. 4, the cam- 
5 era 1 is so constructed as to allow the user to move a cursor 
27 on the display panel 21 and thereby specify his or her de- 
sired position on the screen. In this case, the touch panel 
22 is not needed and can be omitted. 

Also, as another pieces of 3D positional information 

10 needed in presenting the scale image, data about the distance 
between the camera and the object, obtained by utilizing an 
auto- or manual focusing controller built in the camera, may 
also be used. In that case, there is no need to capture an 
image containing the 3D positional information or to provide 

15 the light-emitting means 16 made up of the first and second 
flash lamps 11a and lib and light source controller 12. That 
is to say, any camera can present a scale image on the dis- 
play so long as the camera includes the auto- or manual fo- 
cusing controller and a normal imager that can obtain 

20 information about the distance between the camera and the ob- 
ject • 

Specifically, the real lengths (P^, Py) represented by 
each pixel on the CCD are calculated for a distance L by the 
following equations : 
25 Px = L/f * S„/N^ 
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Py = L/f • Sy/Ny 

where L is the distance data obtained by the auto- or manual 
focusing controller (i.e., the distance between the camera 
and the object), f is the focal length of the camera lens, 
5 and Sy are the sizes of the CCD and and Ny are the numbers 
of active pixels on the CCD. In these notations, the sub- 
scripts X and y denote the horizontal and vertical direc- 
tions, respectively. Then, a scale image is generated based 
on these lengths (P,,, Py) obtained. For example, a scale image 

10 with a length represented by a pixel number N is generated, 
and letters representing the real length X ( in the x di- 
rection) or Ny X Py (in the y direction) are presented on the 
screen. Alternatively, a pixel number R/P, (in the x direc- 
tion) or R/Py (in the y direction) corresponding to a real 

15 length R may be obtained first. In that case, a scale image 
with a length represented by the pixel nuit±)er may be generated 
and then the letters representing the length R may be present- 
ed on the screen along with the scale image. 

Also, the calculation results representing sizes of the 

20 object are preferably stored on the storage medium 25 by the 
recording/reproducing section 24. This is because the user 
does not have to remember the results in that case. Further- 
more, the information stored on the storage medium 25 can also 
be utilized conveniently by using a personal computer, for ex- 

25 ample, with functions equivalent to those of the record- 
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ing/reproducing section 24. It is naturally possible to store 
the range image, the color image or the synthesized image ob- 
tained by combining the scale image with the color image on 
the storage medium 25. 
5 Moreover, the CCD sizes, focal length of the lens during 

imaging and distance data of the object obtained by the auto- 
or manual focusing controller are preferably stored along with 
the image data on the storage mediiam 25 by the record- 
ing/reproducing section 24. In that case, even if no images 

10 are being captured, the scale image can also be read out with 
the image already recorded. 

Also, as shown in FIG. 5, an image memory 35 may be 
provided to temporarily store the image data, received from 
the imager 13, thereon. In that case, multiple pictures may 

15 be input to the camera consecutively and then recorded by the 
recording/reproducing section 24 on the storage medium 25 af- 
terwards. In addition, multiple pictures may be read out from 
the storage medium 25 onto the image memory 35 and then pre- 
sented one after another on the screen at high speeds. 

20 It should be noted that the scale image does not have to 

be presented in the shape of a ruler but may be in any other 
shape so long as the image can be a reference picture repre- 
senting a real size of an object. For example, the scale im- 
age may represent baseball bat, golf club or even a row of 

25 cartoon character face symbols. Furthermore, the user may 
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also select his or her favorite type of scale image or change 
the types of scale images depending on the image to be pre- 
sented. 

Also, according to this embodiment, an image of an ob- 
5 ject may be presented substantially in its real size on the 
display panel 21 by utilizing the real size information 
(i.e., 3D positional information) of the object contained in 
the image data obtained. This is easily realizable by 
providing the length of a single pixel on the display 

10 panel 21 to the image synthesizer 31. Specifically, the num- 
ber Np of on-screen pixels equivalent to a real length X is 
given by 

Np = X/P, 

Thus, the image synthesizer 31 may scale the image up or down 
15 so that when the image is presented on the display panel 21, 
its number of pixels, corresponding to the real length X, be- 
comes equal to the number Np. 

In the illustrated embodiment, the display panel 21 is 
built in the casing 1 and its size is relatively small. Ac- 
20 cordingly, it is impossible to present an image of a big ob- 
ject in its real size on the panel 21. However, if an 
external display panel, e.g., a display for a desktop person- 
al computer, is used, even such an image can be presented 
thereon in its real size. Also, the image, representing the 
25 object in its real size, may be output as a hardcopy using a 
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printer. 
EMBODIMENT 2 

A geometric measuring camera, which is an exemplary im- 
5 age processor according to a second embodiment of the present 
invention, has essentially the same configuration as that il- 
lustrated in FIGS. 1 and 2. But unlike the first embodiment, 
the image synthesizer 31 of the second embodiment combines a 
plurality of images together after at least one of the images 

10 has been scaled up or down according to a user defined size 
relationship. That is to say, according to the second em- 
bodiment, the range images, generated by the range image gen- 
erator 14, are also used as 3D positional information. In 
addition, a synthesized image, obtained by combining multiple 

15 images together, is presented after at least one of the images 
has its/their scale(s) adjusted to meet the user defined size 
relationship based on the 3D positional information obtained. 

Hereinafter, it will be described with reference to 
FIGS. 6A through 6E how the camera of the second embodiment 

20 operates in such a manner as to combine multiple images to- 
gether in accordance with the user's instructions. In the 
illustrated example, the camera shall combine an image of a 
desk with an image of the user's room. 

First, as shown in FIG. 6A, the user takes a picture of 

25 an object (i.e., a desk in the illustrated example) using the 
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camera of this embodiment. Then, the user gets the image 
data (representing the range and color images of the desk 
that have been generated by the range and color image genera- 
tors 14 and 15, respectively) stored on the storage medium 25 
5 by the recording/reproducing section 24, 

At the same time, as shown in FIG, 6B, the user also 
takes a picture of a room in his or her home, for example, 
using the camera of this embodiment. Then, the user gets the 
image data, representing the range and color images of the 

10 room, stored on the storage medium 25 by the record- 
ing/reproducing section 24. 

Next, as shown in FIG. 6C, the user reads out the desk 
image stored and gets the image separated from its background 
image and then presented on the display panel 21. In this 

15 case, an object image can be separated from the background 
image by getting presented only a portion of the captured 
color image that is defined by predetermined three- 
dimensional coordinates. In the illustrated embodiment, by 
using the range image, a portion of the color image, which is 

20 made up of pixels at respective locations associated with dis- 
tances falling within a predetermined range from the camera, 
is cut out as the desk image. For example, portion of the 
color image with distance data of 2 meters or more as repre- 
sented by the range image may be regarded as belonging to the 

25 background. In that case, only an image of the object, which 
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is located less than 2 meter away from the camera (i.e., the 
desk image in this case), can be cut out. 

Then, the user specifies an alignment point a on the 
separated desk image using the touch panel 22. In the illus- 
5 trated example, the user wants to place the desk image at a 
room's wall image portion. Accordingly, the user defines a 
point a at an upper edge of the desk image as a reference 
point to be aligned with the room's wall image portion. 

Subsequently, as shown in FIG. 6D, the user reads out 

10 the room image stored and gets the image presented on the 
display panel 21. Then, the user specifies another alignment 
point /3 using the touch panel 22. In the illustrated exam- 
ple, the user sets the point 0 on that image portion repre- 
senting the room's wall. 

15 Thereafter, as shown in FIG. 6E, the user combines the 

desk image shown in FIG. 6C with the room image shown in FIG. 
6D. These images are combined together in such a manner that 
those two specified points a and 0 are located at the same 
position within the three-dimensional space. That is to say, 

20 a synthesized image, representing a room with a desk placed at 
a corner, is generated so that the point a is aligned with 
the point 0 . 

In this case, however, the desk and room images were cap- 
tured under mutually different conditions. Specifically, the 
25 distance between the camera and one of the objects (e.g.. 
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desk) is different from the distance between the camera and 
the other (e.g., room). In addition, these images were cap- 
tured with the magnification of a zoom lens defined at mutual- 
ly different values. Accordingly, if these images are 
5 combined together as they are, then part of the resultant syn- 
thesized image might not reflect a desired size relationship 
(e.g., not in proportion to their real sizes). That is to 
say, the scale of the desk image might be too large or too 
small compared to the room image in view of their desired size 
10 relationship. 

Thus, according to the second embodiment, the desk and 
room images are combined together in such a manner that their 
scales are adjusted to meet the desired size relationship by 
using the range images. Specifically, this process includes 
15 the following processing steps. 

First, the magnitude of translation (1, m, n) between the 
specified points a (xa, ya, za) and /? (x/?, y j3 , z/3) is 
obtained by the following equations: 

l = x/3 —xa ; 
20 m=Y/3 —y a ; and 

1 — Z0 —za . 

Next, each of the pixels, making up the desk image shown 
in FIG. 6C, is translated three-dimensionally by the magnitude 
(1, m, n) to its associated point on a plane containing the 
25 specified point 0 . Supposing a point with a set of three- 
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dimensional coordinates should be translated to another point 
Pj with a different set of three-dimensional coordinates, the 
points Pi and P^ are represented as : 
Pi=[xl yl 2l 1] and 
5 P2= [x2 y2 z2 1 ] , respectively. 

And the point P^ is given by the following equation 



P2 = Pl 



10 



10 0 0 
0 10 0 
0 0 10 
LI m n 1 



Next, a two-dimensional desk image to be presented on the 
panel is derived from the three-dimensional coordinates of the 
translated desk image through perspective transformation. In 
transforming a point P with three-dimensional coordinates (X, 
15 Y, Z) onto another point Q with two-dimensional coordinates 
(X, y) on a projective plane as shown in FIG. 1, these two- 
dimensional coordinates are given by 

x = f • X/Z and 

y = f • Y/Z 

20 where f is the focal length of the camera. If the desk and 
room images were captured at mutually different zooming magni- 
fications, then these images should preferably be scaled up or 
down so that these images are presented at the same magnifica- 
tion. That is to say, projective transformation needs to be 

25 carried out in such a manner as to equalize the f values in 
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the above equations with each other. 

Thereafter, a texture of the desk surface is extracted 
from the color image and then mapped onto the resultant desk 
image. 

5 As a result of this process, the desk image is combined 

with the room image after their scales have been matched to 
each other as shown in FIG, 6E, An image synthesis process 
like this is performed by the image synthesizer 31. In this 
manner, according to this embodiment, a separated object image 

10 can be combined with a different background image after at 
least one of the images has its /their scale(s) adjusted to 
meet a desired size relationship even if these images were 
captured at mutually different distances or zooming magnifica- 
tions. Accordingly, in a resultant synthesized image, the ob- 

15 ject (i.e., the desk in the illustrated example) looks like as 
if the object had been imaged at a place (i.e., the room shown 
in FIG. 6B) different from the place where it was actually im- 
aged . 

The alignment points a and /? are supposed to be freely 
20 specified by the user. Thus, as a result of the image synthe- 
sis, a non-realistic image, in which the image components are 
placed at impossible positions, might be generated. For exam- 
ple, a leg of the desk might sink into the floor of the room 
or the upper surface of the desk might force itself into a 
25 wall of the room in such an image. Accordingly, in view of 
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these potential unfavorable situations, the relative positions 
of these images to be combined are preferably made definable 
or modifiable externally. For instance, the camera may be con- 
structed so that the user can move the images using the touch 
5 panel 22 or a switch provided on the back of the camera. 

For example, if a desk leg has sunk into the floor, then 
the user should manually specify the magnitude of translation 
toward the top of the screen. In accordance with the user's, 
instructions, the image synthesizer 31 translates the desk im- 

10 age upward, combines the translated desk image with the room 
image again and then gets the resultant synthesized image pre- 
sented on the display panel 21. In this manner, the user re- 
peatedly specifies the magnitude of translation while checking 
back the synthesized image on the screen until the desk image 

15 is placed at a most satisfactory position. 

In the foregoing embodiment, the perspective transforma- 
tion is supposed to be carried out for respective points on 
the desk image. Alternatively, these images may also be com- 
bined by an approximation method. For example, according to a 

20 method, the distance between the point a and the camera and 
the distance between the point /? and the camera may be ob- 
tained from the range images and a two-dimensional scaling 
process may be performed with a ratio of these distances re- 
garded as the magnification of the desk image, 

25 Also, in the foregoing embodiment, at least one of the 
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images is transformed so that their scales are matched to each 
other according to desired size specifications. Optionally, 
the user may intentionally change the scale of one of these 
images so that the image will be out of proportion on purpose. 
5 In that case, the image may be scaled up or down two- 
dimensionally . Or the object images may be scaled up or down 
three-dimensionally based on the 3D data of the images and 
then a resultant synthesized image may be obtained through 
perspective transformation. In other words, the image synthe- 

10 sizer 31 may combine multiple images together in such a manner 
as to meet a desired size relationship. As used herein, "com- 
bining images to meet a desired size relationship" includes 
combining the images together after at least one of the im- 
age(s) has /have been scaled up or down according to a substan- 

15 tially actual size relationship as described above. 

That is to say, the image synthesizer 31 is so construct- 
ed as to up- or downscale at least one of multiple images to 
be combined. In this case, the scaling process may be per- 
formed through the following three-dimensional transformation. 

20 The images to be scaled up or down have the specified points 
represented as: 

Pi=[xl yl zl 1] and 
P2=[x2 y2 z2 1], respectively. 
And the point is given by the following equation 
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P2 = Pl 



kx 0 0 0 

0 ky 0 0 

0 0 kz 0 

0 0 0 1 



5 In this manner, the user can arbitrarily scale up or 

down the image of a particular object. Thus, a miniaturized 
synthesized image can be obtained, for example. 

Also, if the camera tilted during imaging, then the re- 
sultant object image (i.e., the desk image in the illustrated 

10 example) might also tilt. In that situation, the desk image 
should preferably be combined with the room image after the 
upper surface of the desk image has been parallelized to the 
floor of the room image by subjecting the desk image to rota- 
tional transformation. To realize the rotational transforma- 

15 tion like this, the 4X4 matrix in above equation may be 
replaced with a matrix given by the following Equation: 
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10 0 0 
0 10 0 
0 0 10 
-1 -m -n 1 



10 0 0 
0 10 0 

0 0 10 

1 m n 1 



R = 



n* t-d-n'Ocos 0 luny (1-cos 0 ) +nz sinO iixnx (l-cos 0 ) -iiy si\^^ 0 

nxny (l-cos W )-n2 sin 0 n? + ( 1- n?) cos 0 nyiiz (l-cos 0 ) +nx sin0 0 

nxiii (l-cos 0 ) »-ny sinO nyn^ (l-cos 0 ) -lu siii0 + ( 1- n'O cos 0 0 

0 0 0 1 



where R is a rotational matrix. By using this matrix, the 
25 image is rotated by 6 degrees around the axis passing the 
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origin and a point (1, m, n) . Also, n^, n^ and n^ are speci- 



fied as follows: 



n^^cos a 



ny = cos/? and 



5 



n^ = cos y 



where a , 13 and y are angles formed by the axis of rotation 



with the y- and z-axes, respectively. In a special situ- 



ation where the image has been rotated around the x-axis. 



Ti^ — \ and ny = n3 = 0 



10 Or if the image has been rotated around the y-axis , then 
n^ = n^ = 0 and ny = l 
Or if the image has been rotated around the z-axis, then 
n^ = ny = 0 and r\^ = \ 

U By setting n^, ny and n^ to these values, the rotational ma- 

%| 15 trix R can be modified into a simple form. 

-Q Also, as another pieces of 3D positional information 

needed in matching the scales of multiple images to each 
other, data about the distance between the camera and an ob- 
ject, obtained by the auto- or manual focusing controller 

20 built in the camera, may also be used. In that case, there 
is no need to capture an image containing the 3D positional 
information or to provide the light-emitting means 16 made up 
of the first and second flash lamps 11a and lib and light 
source controller 12. That is to say, any camera can approxi- 

25 mately match the scales of the images to each other so long 
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as the camera includes the auto- or manual focusing control- 
ler and a normal imager that can obtain information about the 
distance between the camera and an object. 

In the foregoing embodiment, the image is a natural im- 
5 age to be taken with a camera. Alternatively, the image may 
also be a synthesized image formed by a computer graphics 
process . 

Also, in the second embodiment, two types of images, 
i.e., the desk and room images, are combined. However, ac- 

10 cording to the present invention, three or more types of im- 
ages can also be combined together with their scales matched 
according to desired size specifications if the above process 
is performed on each of these images. In that case, images 
of multiple objects, which were actually captured against mu- 

15 tually different backgrounds, can be combined together on a 
common background image with their scales matched to each 
other. 

In the second embodiment, the alignment points a and 0 
are supposed to be specified using the touch panel 22. Al- 

20 ternatively, a penlike pointing device, allowing the user to 
specify arbitrary coordinates on the display panel 21, may be 
used instead of the touch panel 22. Cursor keys, press but- 
tons, dial or mouse, which allows the user to move a cursor on 
the display panel 21 as shown in FIG. 4, may also be used. 

25 Then, the touch panel 22 is not needed and can be omitted - 
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Moreover, the CCD sizes, focal length of the lens during 
imaging and distance data of the objects obtained by the auto- 
or manual focusing controller are preferably stored, along 
with the image data, on the storage medium 25 by the record- 
5 ing/ reproducing section 24. In that case, even if no images 
are being captured, ijmages can also be combined in an arbitra- 
ry manner while the images are being presented on the screen. 

Also, the light-emitting means 16 is removable. Accord- 
ingly, the camera may be used with the section 16 attached 

10 thereto when range images should be captured. And the camera 
may be used with the light-emitting means 16 removed therefrom 
when normal images should be captured. However, the light- 
emitting means 16 may naturally be fixed to the camera. 

Furthermore, in the first and second embodiments, the 

15 present invention has been described as being applied to a 
geometric measuring camera capturing a range image (i.e., 
depth image) by imaging a part of the light that has been pro- 
jected from the light-emitting means 16 and then reflected off 
from the object. Alternatively, the present invention is also 

20 easily implementable even with a camera utilizing any other 
depth measuring method. For example, the present invention is 
applicable to triangulation in which images captured by two 
horizontally arranged cameras are matched stereoscopically . 
Or the present invention is also applicable to a method of 

25 measuring the time of flight of a laser spot beam that has 
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been projected to sequentially scan a space with time. This 
is because a depth image can be obtained in each of these al- 
ternative applications. In short, so long as two-dimensional 
images and range information can be obtained, the present in- 
5 vention is applicable to any camera no matter how the camera 
collects the range information. 

Furthermore, the application of the present invention is 
not necessarily limited to cameras. Specifically, the present 
invention is also implementable as an image processor with no 

10 imaging section (e.g., a personal computer) if the apparatus 
can execute the same function as that performed by the image 
synthesizer 31. In that case, responsive to a color image and 
image data (e.g., range image) containing 3D positional infor- 
mation, the image processor either presents a scale image or 

15 combine multiple images together with their scales matched 
through the same process as that exemplified in the first or 
second embodiment. Examples of means for inputting user's in- 
structions include mouse, keyboard, track ball, switch and 
volume. 

20 It should be noted that part or all of the functions of 

the inventive image processor may be implemented using either 
dedicated hardware or software that includes a program to be 
executed by a computer. Also, the program, getting the func- 
tions of the inventive processor executed by a computer either 

25 partially or fully, may be recorded on a computer-readable 
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storage medium so that the computer can execute the program 
recorded on the storage medium. 

In the foregoing embodiments, the geometric measuring 
camera shown in FIG. 1 is supposed to take a still picture. 
5 However, even if the camera can take a moving picture, the 
present invention is implementable in a similar manner. In 
that case, a light source that can project two different types 
of light patterns onto an object alternately and successively 
as shown in FIG. 11 should be used though. A light source 

10 like this is implementable as a pair of flash lamps that flash 
alternately and successively- Alternatively, a laser beam, 
emitted from a semiconductor laser diode, may be shaped into a 
linear beam through a rod lens and then have its intensity 
modulated while making the linear beam sweep the object using 

15 a galvanic mirror, for example. It should be noted that if 
moving pictures of objects have been taken, then those images 
may be naturally combined or presented as they are. In addi- 
tion, it is also possible to process a still picture contained 
in the moving picture and then combine the still picture proc- 

20 essed with the moving picture. 

Application to image processing services 
In the foregoing embodiments, the functions of imaging, 
image generation, image synthesis and presentation are all im- 
25 plemented within the casing 1 of the camera. In other words. 
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the camera is supposed to realize all of these functions by 
itself. However, according to the present invention, the same 
effects are also attainable even if these functions are real- 
ized by discrete units with data exchanged between these 
5 units. That is to say, such an embodiment of the present in- 
vention realizes novel image processing services. 

FIG, 14 is a block diagram illustrating the basic concept 
of the inventive image processing services. In FIG. 14, an 
imaging section 61 includes the light sources 11a and lib, 

10 light source controller 12, imager 13, shutter release 23 and 
control unit 30 shown in FIG. 1. First, the user of these 
services takes a picture of an object using the imaging sec- 
tion 61. In this case, the picture taken should be an image 
containing 3D positional information or at least information 

15 from which the 3D positional information can be drawn. Then, 
the user sends out the image, along with his or her desired 
mode of processing or desired specific type of data processed, 
to a process service server 63 through a network connection 
section 62a, thereby submitting a request for image processing 

20 to the administrator of the server 63. In this case, the mode 
of processing means a specific processing method such as 3D 
extract operation or computer graphics (CG) data computation. 
The specific type of data processed means a specific type of 
output data like 3D CG data or 3D stereoscopic image data. 

25 In this manner, the administrator accepts the image 
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processing request via the network connection section 62a and 
the network. Then, the administrator gets range and color im- 
ages generated from the image data, which the user has ob- 
tained using the imaging section 61 and then sent out through 
the network connection section 62a, by the range and color im- 
age generators 14 and 15, respectively. Next, the administra- 
tor makes the image synthesizer 31 perform the image 
processing on the image in the specified mode (or so that the 
specified type of data can be obtained through the processing) 
while taking the size of the object image received into ac- 
count. And the administrator sends out the resultant image 
data to the user via the network. In response, the user 
checks the received image data on a display section 64. 

In this case, the image processing services to be 
provided may take any of various forms utilizing the 3D posi- 
tional information, e.g., presentation of a scale image or 
combination of multiple images as described for the foregoing 
embodiments. Also, not just the image data transmitted from 
the user, but also any other image data may be used for com- 
bination. For example, the image data of one of well-known 
objects (e.g., celebrities) that have been stored in advance 
in the storage of the process service server 63 may be com- 
bined with the image data received from the user. 

In this case, the user should pay the charge for the 
process services to the administrator. The charge may be 
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^ 

either transferred in cash to the server administrator's bank 
account or paid from the user to the administrator by way of 
a credit card company. 

In the illustrated example, the image data is supposed 
5 to be exchanged via a network. Alternatively, if the image 
data is stored on a storage medium, the image data can be de- 
livered to the administrator either by post or at his or her 
shop. 

10 Application to customization 

Also, by utilizing the technique of combining images to- 
gether using the geometric measuring camera shown in FIG. 1 
to meet a desired size relationship as in the second embodi- 
ment, a method of processing an order for a product in accor- 

15 dance with the real size information of the product is 
implementable • 

In the following example, it will be described with ref- 
erence to FIG. 15 how to fill an order for a customization on 
a wristwatch. As shown in FIG. 15, a product catalog server 
20 65 is connected to a network. The server 65 includes a da- 
tabase on which the 3D structural information, color image in- 
formation and price information of various products are 
stored . 

First, the user takes a picture of his or her wrist us- 
25 ing the geometric measuring camera 1. Next, the user ac- 
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cesses the product catalog server 65 via the network, checks 
a product catalog using a browser and selects a wristwatch of 
his or her desired design. In response, the server 65 sends 
an image of the wristwatch of the specified type, containing 
5 the 3D positional information thereof, to the user. Then, 
the user gets the received watch image combined with the 
wrist image by the image synthesizer 31 of the camera 1 ac- 
cording to the actual size relationship as described for the 
second embodiment. As a result, a virtual image, represent- 

^ 10 ing the wristwatch strapped on the user's wrist, is presented 

^ on the display panel 21. 

g Watching the image on the panel 21, the user determines 

=" whether the design or size of the wristwatch fits in with his 

or her wrist. And if the user likes it, he or she submits an 
Si 15 order for the product to an order processing server 66 via 
O the network. 

In this case, if the product is customizable, then the 
user may select his or her most preferred size of the product 
by scaling up or down the watch image presented on the panel 
20 21 using switches, for example. That is to say, in accor- 
dance with the user's instructions, the image synthesizer 31 
identifies the real size of the wristwatch to be ordered 
based on the magnification selected. In this manner, the 
user may place his or her order, along with its real size in- 
25 formation, with the order processing server 66. Thus the cus- 
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tomization is easily realizable over the net. Furthermore, 
not only the wristwatch as a whole but also respective parts 
(e^g., the dial plate, strap and winding knob) of the watch 
may have their images up- or downscaled independently. In 
5 that case, the user may select parts that will fit in with 
his or her wrist best and may place an order for a wristwatch 
made up of those parts according to his or her preference. 
Alternatively, the 3D structural data of the parts may be 
stored in advance on the catalog server 65. Then, the user 

10 may combine the images of his or her favorite parts together, 
present an image of a wristwatch of his or her customized des- 
ign on the display panel 21 and then place a personalized or- 
der for the watch with its parts, color, size, etc. specified. 
The charge for this order may be paid by any of currently 

15 available paying methods. For example, the charge may be paid 
by credit card, transferred to a specified bank account or 
paid when the user receives the product. 

An order processing method like this is naturally appli- 
cable to customization on any other product, not just wrist- 

20 watches. As an example, it will be briefly described how to 
customize a piece of furniture (e.g., a desk). First, the 
user takes a picture of a room in his or her home using the 
geometric measuring camera 1. Next, the user accesses the 
catalog server 65 via the network, views a product catalog 

25 using a browser and selects a desk of his or her desired 
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specifications. Then, the user gets the desk image combined 
with the room image by the image synthesizer 31 according to 
the actual size relationship as shown in FIG. 6E. As a re- 
sult, a virtual image, representing a room with a desk, is 
5 presented on the display panel 21. Watching the image on the 
panel 21, the user may freely change the scale or position of 
the desk using switches. And when the user places an order, 
he or she should send the product ID number of the desk 
_ preferred and the magnification selected or the real size in- 

10. formation of the desk to the order processing server 66. 
% In the foregoing description, the size of a product is 

m supposed to be freely selectable by the user. Alternatively, 

several different sizes may be prepared for a single product 
M= on the receiving end so that the user may select his or her 

M 15 favorite size from these. In that case, the product is not 
O totally customizable but can be of just a limited number of 

sizes. Thus, the seller can cut down on the costs of devel- 
oping products . 

As is apparent from the foregoing description, a scale 
20 image, representing a real size of an object, is combined 
with an original image to generate and present a synthesized 
image according to the present invention. Thus, on looking 
at the image presented, the user can know the real size of 
the object imaged. In addition, according to the present in- 
25 vent ion, multiple images can be combined together with their 
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scales adjusted in accordance with desired size specifica- 
tions (e.g., according to an actual size relationship). Ac- 
cordingly, the user can quickly watch a virtual image of an 
object against a different background, for example. 



